Let the backend test job fail the run - #13827
Open
hysts wants to merge 1 commit into
Open
Conversation
continue-on-error sat at the job level, so it covered all four matrix combinations, including ubuntu + not-flaky. A failing backend job was recorded as failure on its own check run but the check suite and the workflow run both rolled up to success, so the commit got a green tick and only the individual entry in gh pr checks was red. That does not weaken the PR gate, since required status checks read the job's check run, which was already failure. What it hid was pushes to main, where there is no PR to block: 19cb8f7 currently sits on main with a failing backend job and a green tick. Scoping it to the combinations that are expected to be unreliable keeps the original intent. Nothing downstream keys off this workflow's conclusion, and the only failure it would newly expose on main is the one fixed in #13824.
Collaborator
🪼 branch checks and previews
Install Gradio from this PR pip install https://huggingface.co/buckets/gradio/pypi-previews/resolve/fdc56fce172e6925e7cd862c119e75f1e227ee83/gradio-6.26.0-py3-none-any.whlInstall Gradio Python Client from this PR pip install "gradio-client @ git+https://github.com/gradio-app/gradio@fdc56fce172e6925e7cd862c119e75f1e227ee83#subdirectory=client/python"Import Gradio JS Client from this PR via CDN import { Client } from "https://huggingface.co/buckets/gradio/npm-previews/resolve/fdc56fce172e6925e7cd862c119e75f1e227ee83/browser.js"; |
Collaborator
🦄 no changes detectedThis Pull Request does not include changes to any packages.__No changes detected. __
|
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The change is narrowly scoped to CI configuration and correctly limits continue-on-error to the intended matrix variants without impacting the primary test signal.
Pull request overview
Adjusts the python GitHub Actions workflow so the overall workflow/run status reflects failures in the primary backend test job, while still tolerating failures for explicitly “unreliable” matrix variants (flaky suite and Windows).
Changes:
- Replaces job-level
continue-on-error: truewith a matrix-scoped condition. - Ensures
ubuntu-latest+not flakyfailures propagate to the workflow/run conclusion, improving signal onmainpushes.
File summaries
| File | Description |
|---|---|
| .github/workflows/test-python.yml | Scopes continue-on-error to flaky/Windows matrix entries so the main Ubuntu non-flaky job can fail the run. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
hysts
marked this pull request as ready for review
September 5, 2026 02:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
continue-on-error: truesat at the job level on thetestmatrix, so it covered all four combinations, includingubuntu-latest+not flaky. When that job failed, its own check run was recorded asfailure, but the check suite and the workflow run both rolled up tosuccess. The roll-up is what the commit list andgh run listshow, so a red job appeared as a green tick.This does not weaken the PR gate, and the change does not tighten it either: required status checks match on the check run name, and
test-ubuntu-latest-not-flakywas alreadyfailurethere. What was hidden is pushes tomain, where there is no PR to gate.19cb8f7bcis onmainright now with a failing backend job and a green tick, and fourtest_load_assetsfailures reachedmainthe same way until a release PR made them look like release breakage (#13697).The matrix also covers
test-type: flakyandwindows-latest, both of which run only under a label and are expected to be unreliable, which is presumably what the setting was for. Scoping it to those keeps that intent and lets the combination everyone actually reads tell the truth.Two checks behind the change, both done before proposing it:
github.event.workflow_run.conclusion == 'success'and every one listens to a different workflow. The only one listening topythonisupdate-checks.yml, which fires oncompletedregardless of conclusion and whose single job runs only when the python workflow did not run.mainred for unrelated reasons. Job-level conclusions for the 17 most recentpythonruns onmain(2026-08-24 to 2026-09-04): 11 green, 1 red, 5 that skipped the python tests entirely. The one red is the flakiness fixed by Fix flaky backend tests that build a bareTestClient#13824.Best merged after #13824, so the first honest run is a green one.
Closes: #13826
AI Disclosure
We encourage the use of AI tooling in creating PRs, but the any non-trivial use of AI needs be disclosed. E.g. if you used Claude to write a first draft, you should mention that. Trivial tab-completion doesn't need to be disclosed. You should self-review all PRs, especially if they were generated with AI.
🎯 PRs Should Target Issues
Before your create a PR, please check to see if there is an existing issue for this change. If not, please create an issue before you create this PR, unless the fix is very small.
Not adhering to this guideline will result in the PR being closed.
Testing and Formatting Your Code
PRs will only be merged if tests pass on CI. We recommend at least running the backend tests locally, please set up your Gradio environment locally and run the backed tests:
bash scripts/run_backend_tests.shPlease run these bash scripts to automatically format your code:
bash scripts/format_backend.sh, and (if you made any changes to non-Python files)bash scripts/format_frontend.sh